Early support for @console_rule. #6088

kwlzn · 2018-07-10T08:22:55Z

stuhood

Thanks Kris, this looks great. My main surprise was the changes in GoalRunner... I figured that if the v2 pipeline was running, it would be long-completed by then.

stuhood · 2018-07-10T21:07:54Z

src/python/pants/bin/goal_runner.py

@@ -153,73 +170,75 @@ def _setup_context(self):
      context = Context(options=self._options,
                        run_tracker=self._run_tracker,
                        target_roots=target_root_instances,
+                        v2_target_roots=target_roots,


The fact that Context enters into the picture at all here is surprising to me. It seems like the context could/should just always be created after all v2 logic has already completed?

stuhood · 2018-07-10T21:09:25Z

src/python/pants/bin/goal_runner.py

+    global_options = self._context.options.for_global_scope()
+    self._validate_workdir(global_options.pants_workdir)
+
+    # N.B. For daemon runs, console rules execute pre-fork.


I think that my thinking on this was that @console_rules always ran "first"... in the sense that they were sortof a different phase of the PantsRunner that didn't require the GoalRunner at all. Then the GoalRunner could (optionally) be created after they had run.

makes sense. refactored this a bit - let me know what you think now.

stuhood · 2018-07-10T21:15:09Z

src/python/pants/option/global_options.py

+    # Toggles v1/v2 `Task` vs `@rule` pipelines on/off.
+    register('--v1', advanced=True, type=bool, default=True,
+             help='Enables execution of v1 Tasks.')
+    register('--v2', advanced=True, type=bool, default=False,


It took a little while to grok the split here... I think it makes sense though. Would probably want to explain the heck out of it in the help.

jsirois

Found 1 bug to fix, the other two comments are just ideas. I re-read the v2 Frontend HLD doc and these two ideas are decidedly off-plan, especially the Console product idea. So these are ignorable wrt landing on the moon, but otherwise perhaps useful longer term.

jsirois · 2018-07-15T10:07:52Z

src/python/pants/bin/local_pants_runner.py

+        self._target_roots
+      )
+    except Exception:
+      logger.warn('Encountered unhandled exception {!r} during rule execution!')


Missing format call and arg.

jsirois · 2018-07-15T10:37:50Z

src/python/pants/init/engine_initializer.py

+  @staticmethod
+  def _make_goal_map_from_rules(rules):
+    goal_map = {}
+    goal_to_rule = [(rule.goal, rule) for rule in rules if getattr(rule, 'goal', None) is not None]


This getattr gets to the heart of things - it's not about @console_rule's (with the stress on console) - it's about user-facing goal names. One bit of what seems like conflation is @console_rule - which is just a rule made invokable by a user (from the console) - could be called @goal_rule or @exposed_rule or @cli_rule - and Console. You can imagine you'd wan't a CLI-invokable rule that has no (direct) Console interaction.

@foreground_rule is another name that was floated, since it's the "run in the foreground, exclusively" bit that makes it safe to give them console access.

sure, could be called any of those - but didn't feel strongly enough about it in any particular way beyond what we'd called them in the HLD.

I think from an end user approchability/grokability perspective, the @console_rule naming feels clearest to me to imply "a rule that runs with exclusive access to the terminal console". e.g. most new authors won't understand off the bat what an "@exposed_rule" is etc. ultimately, this is all a matter of conceptual overview and framing in the docs tho.

if either of you feel more strongly about the naming, happy to discuss on Slack. we can always change the naming later too, if deemed more appropriate to be referred to as e.g. @foreground_rule. but I like @console_rule, for the moment.

I don't feel strongly about it.

jsirois · 2018-07-15T11:36:34Z

src/python/pants/engine/rules.py

+
+
+def console_rule(goal_name, input_selectors):
+  output_type = GoalProduct.for_name(goal_name)


If Console were a product type, inverting the moon-landing flow, alot of future goals for console handling might fall out a bit more naturally.

A sketch - here IO is installed as a singleton that is opaque with no methods to take advantage of in user code. In order for user rule code to create a Console (or InteractiveConsole (stdin)) or Logger, they must pass in an IO. Behind the scenes it would probably carry a Clock.

@rule(Console, [Select(IO), Select(BuildFileAddresses)], goal='list') def fast_list(io, addresses): """A fast variant of `./pants list` with a reduced feature set.""" console = Console(io) for address in addresses.dependencies: console.print_stdout(address.spec) return console

So now a @console_rule is just a @rule that produces a Console and has a user-facing CLI name (goal='list'). The console would have no handle to real io, it would just buffer data. The engine could then validate and aggregate and/or serialize CLI @rules and Console products, actually driving the Console with one real console.

If the native engine were modified slightly to handle nullary Get and rules returning a generator - ie: feed generator rules until StopIteration instead of assuming a single yielded value, this could be:

@rule(Stream(Console), [Select(BuildFileAddresses)], goal='list') def fast_list(addresses): """A fast variant of `./pants list` with a reduced feature set.""" console = yield Console(Get(IO)) for address in addresses.dependencies: yield console.print_stdout(address.spec)

Where Console.print_stdout returned a Console.

Hm... I think that from my perspective, Select(IO) is approximately what Select(Console) does currently: ie, represents exclusive access to a console.

I'm not sure I see the advantage of the console being an output product when the other guarantees need to be met (exclusive, foreground access). Even in a world where we added back streaming of the console product with Stream(Console), the API would seem to be unnecessarily restrictive given that we know it "owns" the console while running.

yeah, during design discussions I had initially proposed the idea of a Console as buffered virtual console in a similar vein to what you're suggesting (except without the Console/IO split - just wholesale copies) - but we'd decided against that in favor of more direct, exclusive access to accommodate a wider set of rule types to start.

I think for now, I agree with Stu that these are approximately equivalent - but I definitely see this iteratively evolving way beyond this current ~throwaway impl.

stuhood

This looks awesome... much cleaner than before.

Thanks Kris!

@kwlzn

### Problem The ability to run a command continuously as files change has been a long-standing feature request, and something which is implemented in competing tools. It is very useful for low-latency compiler and test feedback. ### Solution Builds atop @kwlzn 's work in #6088 and adds a `--loop` flag which re-runs all `@console_rule`s whenever input files have changed. Adds a covering integration test (and refactors the pantsd test harness a bit to do so). ### Result ``` ./pants --enable-pantsd --v2 --no-v1 --loop list ${dir}:: ``` ... will continuously list the contents of a directory when files under the directory are invalidated.

@kwlzn

### Problem The ability to run a command continuously as files change has been a long-standing feature request, and something which is implemented in competing tools. It is very useful for low-latency compiler and test feedback. ### Solution Builds atop @kwlzn 's work in pantsbuild#6088 and adds a `--loop` flag which re-runs all `@console_rule`s whenever input files have changed. Adds a covering integration test (and refactors the pantsd test harness a bit to do so). ### Result ``` ./pants --enable-pantsd --v2 --no-v1 --loop list ${dir}:: ``` ... will continuously list the contents of a directory when files under the directory are invalidated.

kwlzn force-pushed the kwlzn/moon_landing branch 2 times, most recently from be57d44 to f95c064 Compare July 10, 2018 17:06

stuhood reviewed Jul 10, 2018

View reviewed changes

stuhood mentioned this pull request Jul 13, 2018

[experiment] Frontend Tasks: v1 tasks may request v2 products #5639

Closed

Console rules.

1c08f01

kwlzn force-pushed the kwlzn/moon_landing branch 3 times, most recently from 9d34d32 to eb579d3 Compare July 14, 2018 09:30

Feedback, Refactoring and Test Repair.

859f6ee

kwlzn force-pushed the kwlzn/moon_landing branch from eb579d3 to 859f6ee Compare July 14, 2018 09:30

kwlzn added 4 commits July 14, 2018 02:47

Goal validation.

72c7bf8

Revert help handling move.

4ddf81b

Dual mode validation.

e95164a

Test fixup.

b632289

kwlzn requested review from stuhood and jsirois July 15, 2018 08:00

jsirois approved these changes Jul 15, 2018

View reviewed changes

stuhood approved these changes Jul 15, 2018

View reviewed changes

Feedback.

72e0861

kwlzn merged commit 5f36f15 into pantsbuild:master Jul 17, 2018

stuhood mentioned this pull request Jul 29, 2018

Add a --loop flag, to allow for running continuously #6270

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Early support for @console_rule. #6088

Early support for @console_rule. #6088

kwlzn commented Jul 10, 2018

stuhood left a comment

stuhood Jul 10, 2018

kwlzn Jul 14, 2018

stuhood Jul 10, 2018

kwlzn Jul 14, 2018

stuhood Jul 10, 2018

jsirois left a comment

jsirois Jul 15, 2018

kwlzn Jul 16, 2018

jsirois Jul 15, 2018

stuhood Jul 15, 2018 •

edited

Loading

kwlzn Jul 16, 2018

stuhood Jul 16, 2018

jsirois Jul 15, 2018 •

edited

Loading

stuhood Jul 15, 2018 •

edited

Loading

kwlzn Jul 16, 2018

stuhood left a comment



		def console_rule(goal_name, input_selectors):
		output_type = GoalProduct.for_name(goal_name)

Early support for @console_rule. #6088

Early support for @console_rule. #6088

Conversation

kwlzn commented Jul 10, 2018

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsirois left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood Jul 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsirois Jul 15, 2018 • edited Loading

Choose a reason for hiding this comment

stuhood Jul 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

stuhood Jul 15, 2018 •

edited

Loading

jsirois Jul 15, 2018 •

edited

Loading

stuhood Jul 15, 2018 •

edited

Loading